This is an experimental and fully un-tested branch, which attempts to add build rules for building jj with Buck2, AKA BUCK files. The goal of such a technology would be to have a fully unified and integrated system for development of Jujutsu — no matter the build platform or client of the build system — even as the project expands and includes more and more features, more developers, and possibly languages and tools.

Please note that I don't think this is a branch that should in its current state be merged (or ever will? Assuming all goes well?) While I was partially motivated to unify things in workspace.dependencies recently because it would help here, that wasn't really the main goal. Part of it was to see if it could even remotely work at all; I am happily using buck2 for my own (private) Rust projects, so it was also an attempt to see if I could bring some of that magic over. I'm happy to say it seems promising but I've run into a (big) issue. In the meantime, I am publishing an interm report and PR on it.

If something like this is truly needed one day, I assume that's because Jujutsu will be wildly successful beyond our dreams, to the point that this sort of tech is a requirement, for reasons I'll go over. But maybe there's more to it than just that? This is above all an experiment, but I'll elaborate.

Because this kind of wide-reaching impact this kind of thing can have (on project direction, on developers, on support, et cetera) the remainder of this description is a very long novel about why you might be interested in this (or maybe why you would hate it.)

Note: This patch series shouldn't be taken as any kind of endorsement or commitment to its maintenance. If you're not interested, don't worry, it's not going anywhere. It may break, get rewritten, or otherwise behave incorrectly. Do not taunt Happy Fun Ball.

NOTE: Jujutsu has many Google employees who work with or on it, or are otherwise in touch with the development; they'll be familiar with much of what I'm about to discuss thanks to the use of Blaze/Bazel for their own needs, but for the purposes of pedagogy I'll assume the reader isn't familiar at all. Perhaps Bazel users will find something interesting or even learn something new, too. Bazel specifically will be discussed later.

Prologue

In the beginning — literally, since the dawn of time — there was make. make is actually not one program but many, but all variants of make have a similar idea: to represent the compilation of software, or software artifacts, as a directed acyclic graph — where each node represents an "action" to perform that results in outputs, and where most actions (non .PHONY ones) can be cached in the filesystem.

make is a good tool and I have a fond spot for it. However, in practice, it is very difficult to write correct Makefiles that work, at scale, for large software projects, without exceedingly austere requirements on the layout of the project, its language, and its specific design points — and while having something that is portable and easy to understand.

For many years many programming languages, including high level ones, used Make (including my beloved Haskell) to drive software compilation; though at some point around the turn of the millenium, it seems engineers began to move away from this model when dealing with high level languages (C/C++ users continue to wallow in misery to this day). There are some decent reasons for this that motivated it, I believe, when you back up a bit:

• Again, make isn't one thing, but a family of things. There is gmake, bmake, there used to be imake, and there is also the venerable mk from plan 9 (which unlike the others actually had the fortitude to fix one of makes worst sins — its interaction with shell escaping in the body of a rule.)
• Even then, make is not just a "build engine", but it's also a DSL and an executable tool. Integrating it is awkward in some ways and often requires contortions like generating fragments of makefiles based on information about the environment. The principles are actually simpler (toposort a dag, check its edges are valid, cache output files by mtime) to reimplement than to piggyback on tool with an impedence mismatch.
• On that note, Make has awful, awful syntax, and this is one of its bigger sins (though other people might disagree), especially its interaction with the shell and how escaping works inside rule bodies.
  • Fun fact: mk solves this by simply never doing any substitution or interpolation in the body of a rule. It writes the body of a rule, literally, to a temporary file, and then executes it while setting environment variables that are set by the mkfile. This was figured out in 1991 or something, but we've suffered with the alternative every day since.
• Make has poor abstraction capabilities, and not only due to its bad syntax; the macro layer for make is typically nothing more than a nightmarish hell of literal in-place expansions, that must respect syntax rules, and which use dynamic scope, not lexical scope.
  • Imagine if the C preprocessor (dynamically scoped, substitute-in-place) was put into a language where you can't even declare a variable ahead of time. That's going to go really well over time, I'm sure!
• Make is simultaneously too low level and too high level to capture many interesting build patterns.
  • For example: make is too low level because it is way too easy to write incorrect rules, and rules leak all kinds of details about them throughout the codebase (such as the need to set global variables based on the body of the rule)
  • It's also too high level because it can't capture important patterns like "dynamic dependencies"; Make often wants to build the entire build graph immediately and start executing, but at scale this is impractical — some things need to be discovered on demand. Make doesn't offer the proper tools to do this. For example, you might generate a .c file from another file, but then you need to scan the .c file for .h files it includes, in order to get a correct build graph. Because make can't dynamically discover this information, you have to repeat the logic of the generator program into the build rules, which is tedious and error prone; you can easily mis-synchronize the generator program and the rules. There is no reuse possible.
  • These aren't theoretical; my former colleague Duncan wrote about the problems with make almost exactly 15 years ago to date, and it's just as relevant now as it was then.
• These things all dovetail to make it really awkward to handle advanced programming language features through make alone, among others. For example, good luck handling C++20 modules. Even for simpler cases, capturing module dependencies is often a bunch of work in Make; for Haskell you need to make sure the compile emits dependency information that is .include'd by Make, similar to gcc -MMD in C or C++.

This is ignoring the packaging and distribution situation for any given language, which are often handled by the build tool too, muddling their responsibilities. But ultimately, I think so many language-specific dependency managers were designed for the above reasons, among many many others — Make just doesn't fit well into many of the patterns modern languages enjoy, and so most decided to go their own way and reimplement the good bits — parallel, toposorted, caching DAG — on their own.

What's the modern standard for language-based build systems?

In my mind, a lot of the "modern" build tools we enjoy sprouted from the seeds originally planted by solutions like pip for Python and rbenv/bundler for ruby. Those tools had a simple insight: every project should have their own list of dependencies and needed tools, and they should be provisioned for that project correctly in an isolated manner.

This was a big step up for language specific tools that make also couldn't match. In practice, the development velocities of these communities became discoupled from traditional package managers in the FOSS world where making sure every dependency X worked with every other dependency Y became difficult. There are other cultural reasons for this, but a big motivation for the "isolation" concept came from that.

Actually, among the modern alternatives, I think rustup and cargo for Rust, as well as ghcup and cabal in Haskell, are about the best examples for modern standard, and I give the edge to cabal somewhat there. (It's not surprising, by the way, that one of the creators of Bundler, Yehuda Katz, was also one of the creators of Cargo.)

But the language-specific-ness remains. Every tool must implement its own build engine, must integrate with language specific features, and has their own distinct UX and lexicon. And those tools also tend to miss out on the greater picture or lack advanced features...

OK, so what's the sales pitch for buck2?

From its own README.md (which was written by yours truly):

• Fast. It doesn't matter whether a single build command takes 60 seconds to complete, or 0.1 seconds: when you have to build things, Buck2 doesn't waste time — it calculates the critical path and gets out of the way, with minimal overhead. It's not just the core design, but also careful attention to detail that makes Buck2 so snappy. Buck2 is up to 2x faster than Buck1 in practice. So you spend more time iterating, and less time waiting.
• Hermetic. When using Remote Execution, Buck2 becomes hermetic: it is required for a build rule to correctly declare all of its inputs; if they aren't specified correctly (e.g. a .c file neeads a .h file that isn't correctly specified), the build will fail. This enforced correctness helps avoids entire classes of errors that most build systems allow, and helps ensure builds work everywhere for all users. And Buck2 correctly tracks dependencies with far better accuracy than Buck1, in more languages, across more scenarios. That means "it compiles on my machine" can become a thing of the past.
• Multi-language. Many teams have to deal with multiple programming languages that have complex inter-dependencies, and struggle to express that. Most people settle with make and tie together dune to pip and cargo. But then how do you run test suites, code coverage, or query code databases? Buck2 is designed to support multiple languages from the start, with abstractions for interoperation. And because it's completely scriptable, and users can implement language support — it's incredibly flexible. Now your Python library can depend on an OCaml library, and your OCaml library can depend on a Rust crate — and with a single build tool, you have a consistent UX to build and test and integrate all of these components.

Why use a large-scale integrated build system like this? What becomes possible? What becomes easy?

Assuming this all worked out, and everyone was on board, here's an example of a few things this could offer us.

• A unified build cache for all developers. One of the best features of these build systems that is immediately visible is that they support caching intermediate build artifacts, often over the network. Buck2 takes this to the biggest extreme it can; nearly all rules it implements can have intermediates cached, it encourages developers to contribute to the cache (not just CI), and it can use pre-existing Bazel-compatible caching services.
  • Example: Any developer can write patches and push a branch into the main jj repo. Any other developer can run jj git fetch and then check out those changes. They run buck2 build and their build completes instantly, thanks to cache hits.
  • Because this works for every commit and every branch, you can say goodbye to ever needing to compile https://crates.io dependencies!
• Remote build execution. Another extreme example of what these tools can do is that, with correctly written BUCK files, all commands can be executed remotely on remote systems in a correct way. This makes many distinctions between e.g. platform-specific development much simpler, because alternative platforms fail in your local development loop just like they would in CI.
  • Example: Any Jujutsu developer who works on Linux can instantly build and run tests on macOS or Windows, all from the same unified build UX (assuming those machines are available to them).
  • Nit: Remote execution does allow nearly infinite build scale when done correctly, so it's really a scale/performance improvement — but for us, I think remote platform builds would actually be the bigger benefit in practice.
• Unification of CI and local development. The previous two points (caching, remote execution) make many of the distinctions between CI systems and local development nearly obsolete. The CI and CD systems can use exactly the same resources developers do.
• A powerful, unified build language. This is actually the most unspoken benefit of such a design: a single build language and UX for building anything. Building docs, building Rust code, building C code, building web pages — they all exist behind buck2 build, using BUCK files that can be programmatically queried and introspected.
• A separation between implementation and interface. In Buck, a rule is just a function call, where the body of the function call is abstract. That means you can change the implementation to your hearts desire, as long as the contract it exposes is preserved.
  • In contrast, systems like Make tie the definition of the rule and the body of the rule together. Due to previous deficiencies, this often means that rule dependencies are dependent on the implementation of rules. Want to refactor a rule? Get ready to change all the dependents.

A good example of this is that in large scale projects, multi-language support becomes essentially necessary, and by then abstraction capabilities are really useful.

For example, the above generality allows much deeper levels of code generation than most tools possibly allow. We can have Rust code depend on a code generator written in Python that depends on C++ that depend on Rust libraries, all with a unified UX. For example, we could automatically generate web pages, man pages, and documentation indicies from the source code itself with techniques like this.

A more concrete example? A theoretical jj web that I keep talking about on Discord. We might have Typescript code that uses React that depends on HTML and hell, maybe all that TypeScript also depends on Rust compiled to WebAssembly. The sky is literally the limit in the webdev world. And we need the Rust code to depend on it, as a distributable. We need to run linters, formatters, and run test suites on that code just like we do with Rust, today to the same standards and quality. And all that can be achieved with a unified UX that behaves the same way. How do you build the HTML frontend? buck2 build. Testing? buck2 test. Running the server with every change? buck2 run.

Interlude: Cargo woes

One of my biggest constant issues with Cargo is that it does not do truly global content-addressable caching. What this means is that you can get into a scenario where things are spuriously recompiled. For example:

• You have a dependency on x=1.2.3
• You build x, resulting in an rlib
• You enable feature foobar on x=1.2.3 and build again.
• The new feature requires recompilation of x, as expected.
• You now disable feature foobar, and build x again
• Cargo will recompile x without foobar, even though it already did that once before

In other words, even though the content of the final build step above is the same as a previous result that was built, Cargo didn't know that and rebuilt it instead. This in turn means that sharing Cargo directories is not possible between projects, because one project may require serde_json with another one requires serde_json with another feature. At a very high level, Cargo doesn't identify things by content, but by name; therefore there can only be one x at a time, regardless of features.

In contrast, Buck2 correctly handles the above scenario, so the second recompilation of x without foobar will be a no-op and complete instantly. Actually, Cargo seems to really like spuriously recompiling things for many reasons beyond my comprehension; I assume this is because it is being safe and conservative to ensure the build succeeds, likely for reasons like this.

You might say "this seems theoretical", but it isn't! It's a specific case of a general problem, and it happens all the time. Anytime you jj edit any new commit, you may have a completely new set of dependencies and versions that can come from a downstream changing. For example, switching between two branches where one has a new feature will cause spurious dependency recompilation, like above.

NOTE: Earlier, I said between the modern alternatives of cargo (Rust) and cabal (Haskell), I gave the edge to cabal — and this exact example is why. Modern cabal based projects use a single globally available build cache that works across all projects, and avoids spurious recompilation like the above. For compilation-heavy languages like Haskell (and Rust), this results in huge disk, I/O, and build-time savings.

... And what becomes hard? (OpenSSL edition)

Dependencies, mostly. One of the drawbacks of the approach tools like Buck, Bazel, and even Nix make is that they truly want the most complete view of the build graph as is possible, and this means they want to understand all inputs and outputs to the system in a deterministic way. Tools operating under this assumption are few and far between e.g. too many programs write files or talk to the network or do any number of other things under the hood in order to be more useful. A simple corollary to keep in mind: small conveniences like these sleights-of-hand actually cost a lot in the long run.

In this version of the design, what this means is that Cargo crates are compiled by Buck, and not really by Cargo. These are automatically generated by a tool called reindeer, which translates Cargo.toml files into BUCK files — effectively translating the static build graph from Cargo to Buck.

This is fantastic in general, but there's a small nit — Rust build.rs scripts, which can range from doing things like "Compile 1 million lines of C++ code on demand" to "Write the current HEAD revision to a file." In general, build.rs scripts are simultaneously:

• Way, way too powerful, as they can do nearly anything and depend on any ambient effect or input,
• Extremely opaque, because their details are not exposed to the user of the build system,
• Really really useful for a wide array of things.

Because of that, and because of the impedence mismatch between Cargo and Buck2 — this patch series contains a commit called buck: add initial reindeer scaffold for rust compilation which adds approximately ~90-100 lines of YAML files in order to work around these sorts of problems; the YAML files are there to inform reindeer about what kind of build.rs scripts the crates in question are using, so it can properly emulate some of their features.

And, right now, this Buck2 build actually is broken right now, and guess why? Because openssl-sys has a build.rs script that leans into the former category — of trying to build and vendor openssl itself and compile a ton of C++ "out of band". I haven't worked a way around this yet, but it is a hard dependency due to the need of openssl by libgit2. Therefore, if this cannot be worked around, this kind of dependency would effectively prevent Buck from ever being used for Jujutsu, until we did something like swith to Gitoxide (as the pure-Rust nature alleviates many of these issues and makes many build scripts simpler; C/C++ compilation is a number-1 cause of many woes like this.)

This is something we take for granted as modern developers; dependencies are easier than ever to add and rely on, but in some sense, we also do not perceive these things as a kind of power that also restricts us and binds us. This is a form of path dependence. This is one of the ways that it can come back to stop promising experiments or alternative avenues dead in their tracks. It's worth keeping in mind.

For the most part, I genuinely find the buck2 workflow to be pretty smooth and far less corruption/recompilation prone than the equivalent Cargo workflows, ignoring bits like IDE integration.

NOTE: I feel it is important to mention that dependencies are something you are responsible for because ultimately, you ship them to your users. I feel Jujutsu should have a very high standard of quality for the dependencies it chooses to rely on; a version control system is one of the most important and vital tools in the entire stack. And personally, I think we do have good standards; the dependency list is relatively small, controlled, and well tested. openssl I think is the only real problematic thing, it's only problematic here in this PR, and there are discussions about gitoxide in the future. The Rust community has very high quality in general, I will add.

So in some ways, the inherently higher demands something like Buck2 places on dependencies might not only be viewed as a limitation, but also a feature and strength; like all things, there is tension in engineering something for and by humans.

Why not Bazel? Could we use it?

Actually, I think Bazel is pretty cool. But I have little experience with it; I'm using Buck2 because I think it's a good project, written by talented people, and I trust the pedigree it came from, and I think many of its design choices mean it actually might be able to give Bazel a true run for its money — whereas many alternative solutions either seem to be catching up or do not match its functionality, Buck2 seems to be hitting all the right high notes and offers many things Bazel itself hasn't or can't.

• Rules in userspace. Bazel rules for a language are often written by third parties, except for one set of libraries: the C/C++/ObjC, and Java rules, all are written in a special variant of Starlark and built into Bazel. These rules cannot be modified or rewritten.

  In contrast, all build rules for all languages in buck2 are written in Starlark in "userspace"; the buck2 binary knows nothing about any programming language. Morally, it's a binary that only knows how to build a graph of commands, and run them (possibly remotely.) This makes things like atomic upgrades between versions, rollbacks, and many other long-tail scenarios like patching the build rules very easy in practice.

• Buck Extension Language, or "BXL". An entire extension framework built on top of the build engine, which can be driven by Starlark, allowing code-driven introspection, automation, and querying. This makes features like LSPs, code search indicies, and other "related" tooling significantly easier to develop and support e.g. you can feasibly develop your own bxl-based LSP integration for rust-analyze or TypeScript.

• Virtual filesystem support. While this is minor and tangential, virtualized jj repositories have been a recurring topic; Buck2 includes direct support for them by way of watchman and integration with Sapling's EdenFS; I don't believe there's any reason we couldn't extend that to Jujutsu as well. I think that makes this a great dogfood opportunity, and a chance for more "vertical" integration between them. I think this could be super exciting if executed correctly.

• Native binary. Again, minor — I don't think it's a huge deal in our case, and I don't care too much about things like GC, the fast startup times from Buck2 being a native executable written in Rust is really wonderful.

As for the could? Well, I suppose. But I don't know how to juggle it, so I can't write it or maintain it.

Should this happen later, instead of now?

Initially it might seem like this is jumping the gun — if we wanted all those great features, we could just wait later for them, right? What's the issue with doing it years down the line when Jujutsu has 20 paid developers and +1 Jujillion installations worldwide?

But there's the rub: by the time you need a thing like this, you are presumably hitting the walls of technical debt that make moving forward on difficult obstacles nearly infeasible, and where existing human-scale constraints make "lateral movement" to new tools like this extremely difficult. It is actually easiest to adopt a tool like this early on, to the most aggressive degree possible, and thus avoid the conundrum entirely, if you believe that you will need it.

A good metaphor is trekking through an unknown jungle. If you walk 10 miles and realize you took the wrong turn 8 miles back, you can of course just begin the long trek back, and take the better path. But it cost you a lot of time. And surely wasn't an easy choice to make — unless the alternative was another 150 miles.

Luckily, the actual code needed to support the build right now is actually pretty darn small if you ignore the prelude. Keep reading for more details on that.

And what about Cargo?

I actually don't think Cargo should be deprecated, if possible, at least not for a very long time — because especially its IDE support through rust-analyzer is fantastic to have for now. Assuming that can be handled (not easy, but also not supremely hard), I think many other problems could be solved; for example, we could invert the whole process, and auto-generate Cargo.toml files from BUCK files instead of the other way around; then use those in crates published on https://crates.io

Or not. I think we could actually get pretty far and a lot of nice bonuses while supporting both. Ideally we would settle on one eventually, I guess.

Should this happen at all?

Maybe not. There are good arguments that Jujutsu might never outgrow the need for something like Cargo, and that the costs of adopting Buck would be wasted — for example, it's just OK to have some manual scripts or repeating yourself a few times, where Buck would allow abstractions. On the other hand, in such a case where you never need its power, Buck would likely never impose many burdens to maintain, so removal would be easy.

A good example of this is again the jj web example before. How do we drive typescript, react, all those tools? What if we wanted those code generators to build e.g. option or command-line-flag indicies for a website? We'd probably have to write a bunch of .bzl code ourselves to do it, in a robust way. So, it's a bunch more work for us, though ostensibly we could make it more widely available, too. But that's a lot more work than a Makefile that's 30 lines and mostly works (except when it breaks.)

Ultimately though it does hinge on whether or not those advantages seem big enough in the grand scale of things. If you're not going to lean into a tool like Buck, and it only ever plays second fiddle to a tool like Cargo, you'll never truly reap its advantages. Doing so requires real commitment, and it is not clear to me if this commitment is the right choice.

So what's the catch? And the risks?

Plenty, right now. Major drawbacks:

• Limited to users of nix develop exclusively
• No testing support
• No simple profile configuration support (e.g. release vs debug)
• No rust-analyzer support (though, it's a high priority for the upstream project, since it's also written in Rust)
• Windows is definitely MIA here, and would need a few questions answered to get it running. (Windows support is definitely a big priority for Buck2 in general, though.)
• Probably many other nits

Warts include:

• Committing about 90,000 LOC from buck2-prelude into the repository is, uh, a choice.
  • The reality is much rosier beyond that, though; there's only about ~200 lines of "real" code added here, and half of that is duplicating Cargo.toml, so it's really only about 100 lines:

austin@GANON:~/src/jj$ wc -l \
    BUCK lib/BUCK cli/BUCK \
    buck/third-party/rust/Cargo.toml \
    buck/.gitattributes buck/update-prelude.sh.in \
    .buckconfig buck/toolchains/BUCK 
  10 BUCK
  39 lib/BUCK
  10 cli/BUCK
  83 buck/third-party/rust/Cargo.toml
   2 buck/.gitattributes
  15 buck/update-prelude.sh.in
  26 .buckconfig
  25 buck/toolchains/BUCK
 210 total

• Duplicating Cargo.toml for Reindeer, even after I went through Herculean *cough* efforts to get rid of that this week.

Risks include:

• Impacted development velocity.
  • The need to more carefully evaluate things like dependencies and build architecture means that development is slowed, as these considerations are more paramount and cannot be ignored.
  • This then switches many operations like adding dependencies from an "optimistic strategy" to a "pessimistic strategy", to lift and abuse a term from the world of concurrency control.
  • Many developers do not care about these things, and may see the inability to e.g. add any dependency as an impediment.
  • The lack of these things tends to result in some NIH syndrome. Sometimes that's good, and sometimes it's bad.
• Continuous maintenance and sunk costs.
  • If all the savings you have are blown on the time you spend hand-holding the tool — multiply that by N developers, and you're deep in the red pretty quickly. (This means the final product needs to have high quality across the board.)
• Higher complexity.
  • It is undeniable that Buck2 is a more complex tool to handle than Cargo, and there is nearly no argument about this in my mind, even if Buck2 takes a stance that much of that complexity is inherent to the problem space.
  • This is also a felt more acutely by those downstream of the project, e.g. those people who package jujutsu.
• People will complain about you on internet forums and internet posts because they don't like that you have gone off the paved road.
  • I actually think this isn't without merit, in a broad sense (some specific complaints may be bogus),
  • But on the other hand we might have some of this already; e.g. jj developers are, you know, expected to use jj to contribute. Maybe not for typofixes, but I also wouldn't expect most typofix patches to need Buck, either.
  • I think providing things like a Cargo build helps ameliorate those concerns.

Can I use this branch right now?

No. The build is currently broken, because I do not have the ability to work around the openssl-sys crate being broken. This is just a WIP snapshot so it does not get lost. But one day, you'll be able to build and run jj-cli like so, perhaps:

$ buck2 build //:jj-cli
$ buck2 run //:jj-cli -- ...

What's the goal of publishing this?

Because I don't want to lose my work, and I would be interested in what people think about all this. I suspect there are a lot of opinions from "sounds awesome" to "OK" to "I'm not convinced".

Will you keep working on it?

Maybe. Actually, the current total time on getting this branch as far as I did took less time than writing up this PR description. So it's not like I've put unbelievable effort into it.

And, here's the thing: I'm committed to Buck2 for the long run for my projects because I do believe it solves many chronic issues I face, to a much greater and more refined degree than its predecessors. Similarly, after a lot playing for the past few weeks, I firmly believe I'm also committed to Jujutsu for the long run, for the same reasons: because it works like how I think a tool of its class should, and does so far better than the competition — it's 10x better for me, not 1x or 2x. You might think of them like peanut butter and jelly, in my opinion. So I'm happy to see them go together, too.

So I might continue this, returning to it occasionally when it's promising, and if it's considered intruiging by the fellow developers who might also be miffed by some of the above issues. It can't be merged in the current state, I think, at least until full builds work. And assuming it went well I can see myself maintaining it for a good long time and documenting it and helping out any/all issues with it. But if nobody is interested, or it's only a resounding "meh", I'll probably shelve it for a good while until its turnkey or I've established my own changes across the codebase more, or something.